Automated test item generation system and method

ABSTRACT

A method and system for using a natural language generator for automatic assessment item generation is disclosed. The natural language generator includes a document structure generator that produces an abstract document specification defining a structure for an assessment item based on user input. The abstract document specification is input into a logical schema generator, which produces a logical schema specification that creates a more detailed specification for an assessment item. Finally, a sentence generator receives the logical schema specification and creates natural language for the assessment item based on the variables defined in the logical schema specification.

CLAIM OF PRIORITY

[0001] This application claims priority to U.S. provisional patentapplication No. 60/461,896, filed Apr. 10, 2003, entitled “AutomatedTest Item Generation System and Method,” which is incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

[0002] The present invention relates to methods and systems forgenerating assessment items automatically. More particularly, thisinvention relates to a computer system and method for providing aninterface for automatically generating word problems for use in testing.

BACKGROUND OF THE INVENTION

[0003] The use of Automatic Item Generation (AIG), the practice ofcreating assessment items algorithmically, is of increasing interest toeducational assessors. AIG permits educational assessors to quicklygenerate assessment items and at least partially automate the generationof such assessment items. Furthermore, assessors can provide differingassessment items, each having a similar difficulty level, to improvetest security. Moreover, adaptive testing having assessment items thatvary systematically in difficulty can be produced. Such advantages haveencouraged the research and development of AIG technologies.

[0004] However, AIG depends upon well-founded models of the cognitiveabilities underlying performance. Where such models are lacking, AIG canonly have heuristic usefulness. Conventional AIG research has beenperformed in areas where well-founded cognitive theories support thedevelopment of AIG algorithms, such as matrix completion and analyticalreasoning. Such areas are generally restricted in content and highlystructured.

[0005] In contrast, progress in AIG of verbal item types has been morelimited, due to the openness of content and the considerable complexityof natural language. In open-ended verbal items, a strong preferenceexists for developing naturalistic items based upon actually publishedmaterials, and the most productive approaches have focused uponproviding techniques to support test developers by supporting moreefficient item selection and evaluation.

[0006] Where constrained item types have required natural languagegeneration, the treatment of verbal materials has been straightforwardand generally uses verbal templates to generate items. Typicaltemplate-based natural language generation includes two salientproperties: 1) a list of phrases or sentences with open slots; and 2)the random or pseudo-random insertion of words from predetermined listsinto particular slots. Template-based generation has the advantage ofbeing straightforward, quick and dependent upon existing items. However,AIG from such simple templates is clearly limited because naturallanguage complexities cannot be captured within a template format.Moreover, since the strings manipulated by template-based systems haveno theoretical status, they do not support any principled analysis ofthe language employed in any particular problem type.

[0007] Conventional template-based AIG systems suffer from four distinctlimitations: maintainability, output flexibility, output quality and aninability to easily produce multilingual outputs. In a template-basedsystem, a large number of lists are stored and manipulated in order togenerate textual output because each list is task or field specific.Accordingly, repetitive lists may be required to complete populate alltask sets.

[0008] In addition, as the number of templates in a template-basedsystem grows, it is more likely that the variety of templates disguisesthe systematic combination of a much smaller set of variables.

[0009] Moreover, systems must resolve context-dependencies inherent inlanguage, such as subject-verb agreement, selection restriction (i.e.,one drives a car, but flies an airplane), definite-indefinite selection(i.e., a student or the student), and the like. Such dependencies arehandled ad hoc in a template-based system.

[0010] Finally, in order to produce a multilingual template-basedsystem, a system maintainer must generate new templates for targetlanguage. Moreover, dependencies between templates and dependenciesbetween entries in templates must be redefined for each template and/orcombination of entries in the target language. As such, significanteffort must be expended and significant resources must be dedicated tocreate a multilingual template-based system.

[0011] What is needed is a method and system for improving conventionalautomatic item generation by using non-template-based algorithms forgenerating assessment item text.

[0012] A need exists for a method and system for improving themaintenance of automatic item generation systems.

[0013] A further need exists for an automatic item generation system andmethod that increases output variety.

[0014] A still further need exists for an automatic item generationsystem and method that produces higher text quality.

[0015] A further need exists for a method and system of automatic testgeneration that more easily permits multilingual textual output.

[0016] The present invention is directed to solving one or more of theproblems described above.

SUMMARY OF THE INVENTION

[0017] A Natural Language Generation (NLG) system according to thepresent invention may be used to perform automatic item generation. TheNLG system may use computational linguistic techniques to automate textgeneration. The NLG system may receive input and determine textualcontent from the input. The textual content may be abstractly formulatedwithout reference to details at the level of sentences, phrases orwords. One or more sentences may be planned based on the abstractcontent. Following this process, the exact wording for each sentence maythen be formed.

[0018] Thus, a major difference between an NLG system according to thepresent invention and a template-based system is that the specificationof textual content is separated from the details of the wording.Building an NLG system entails constructing a model specifying possiblecontent and separating it from a model specifying how that content ismost naturally expressed.

[0019] An NLG system of the present invention is more maintainable thana template-based system because a large number of lists need not bestored and manipulated in order to generate textual output. In anembodiment, the present invention separates knowledge about language ingeneral from knowledge about specific tasks. Accordingly, the presentinvention is easier to maintain and modify to suit changing assessmentneeds and demands.

[0020] In addition, the output of the present invention may be moreflexible than a template-based model. The present invention may be usedto reduce variety between template-based lists in order to enable moreflexible test generation.

[0021] Moreover, systems must resolve context-dependencies inherent inlanguage, such as subject-verb agreement, selection restriction (i.e.,one drives a car, but flies an airplane), definite-indefinite selection(i.e., a student or the student), and the like. Unlike a template-basedsystem, the present invention may include modules that resolve suchdependencies automatically.

[0022] Furthermore, the present invention may permit text generation inmultiple languages more easily than a template-based system. In anembodiment, low-level language-specific details are segregated intolanguage-specific modules, which permit transposition into high-levelknowledge without requiring a complete system redesign. Instead,adapting an NLG system for a target language may merely requireadjustment of the language-specific modules to describe a new set oflinguistic, grammatical, and/or semantic rules and translation of theentries in the template.

[0023] Considerable synergy between AIG within assessment theory and NLGwithin computational linguistics exists. Both AIG and NLG permitautomatic generation of verbal material. Moreover, the verbal materialis generated in complementary ways. While AIG focuses on controllingconstruct-relevant content, NLG focuses on lower-level document details.

[0024] Another advantage of the present invention is the ability todistinguish between factors that significantly affect item difficulty,known as radicals, and factors that do not, known as incidentals.Conventionally, no a priori way of determining radicals and incidentalsis known. In general, the determination of radicals and incidentalsdepends on the cognitive content of the assessment item and the task theassessment item requires respondents to perform. However, in the presentinvention, the vast majority of variables are incidentals, such as theparticular choice of words, grammatical constructions, and phrasingdetails.

[0025] Verbal tasks may be separated into two types of elements: thosewhich involve decoding (i.e., determining the meaning of text aswritten) and those which involve content manipulation (i.e., performinginference or other thought processes on the content after decoding). Thepresent invention specifies how to encode what test subjects decode, soa direct relationship exists between the encoding of the presentinvention and decoding by a respondent. In other words, radicalsgenerally involve content manipulation and wording typically interactswith radicals at a content level.

[0026] The present invention is directed to a method and system ofgenerating tasks that are drawn from a constrained universe ofdiscourse, are susceptible to formalization, require little complexinference, and for which basic verbal comprehension (decoding/encoding)is not the tested subject matter. Such tasks may include, for example,mathematical word problems. Such mathematical word problems may includedistance-rate-time problems, interest rate computation problems,taxation problems, production problems, physics problems, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Aspects, features, benefits and advantages of the embodiments ofthe present invention will be apparent with regard to the followingdescription, appended claims and accompanying drawings where:

[0028]FIG. 1 depicts an exemplary high-level implementation of a systemfor performing natural language generation according to an embodiment ofthe present invention.

[0029]FIG. 2 depicts an exemplary screen shot of an existing wordproblem assessment item according to an embodiment of the presentinvention.

[0030]FIG. 3 depicts an exemplary variable modification screen for adistance-rate-time assessment item according to an embodiment of thepresent invention.

[0031]FIG. 4 depicts an exemplary detailed control screen for adistance-rate-time assessment item according to an embodiment of thepresent invention.

[0032]FIG. 5 depicts an exemplary task-relevant problem structure for anassessment item according to an embodiment of the present invention.

[0033]FIG. 6 depicts an exemplary measurement unit definition screenaccording to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Before the present structures, systems and methods are described,it is to be understood that this invention is not limited to particularstructures, systems, methodologies or protocols described, as these mayvary. It is also to be understood that the terminology used in thedescription is for the purpose of describing the particular versions orembodiments only, and is not intended to limit the scope of the presentinvention.

[0035] It must also be noted that as used herein, the singular forms“a,” “an” and “the” include plural references unless the context clearlydictates otherwise. Thus, for example, reference to a “variable” is areference to one or more variables and equivalents thereof known tothose skilled in the art, and so forth. Unless defined otherwise, alltechnical and scientific terms used herein have the same meanings ascommonly understood by one of ordinary skill in the art. Although anymethods, devices and material similar or equivalent to those describedherein can be used in the practice of testing of embodiments of thepresent invention, the preferred methods, devices, and materials are nowdescribed. All publications mentioned herein are incorporated byreference. Nothing herein is to be construed as an admission that theinvention is not entitled to antedate such disclosure by virtue of priorinvention.

[0036] The term “semantic frame,” as used herein, refers to a method fororganizing verbal content. In particular, a semantic frame makes itpossible to analyze a particular type of word problem and isolate aseries of variables with clear task-model relevance.

[0037] Semantic frames may be organized as a hierarchy of frames. Eachsemantic frame may contain linguistic detail including the schematicstructure and the vocabulary associated with the particular semanticframe. Semantic frames may be used to denote particular patterns ofvocabulary for certain types of assessment items that are highlysystematic and have constrained vocabulary and syntax, such asmathematical word problems.

[0038] The term “schematic logical representation,” as used herein,refers to information contained in a file pertaining to the syntax andvocabulary used to generate natural language for a given assessmentitem. A schematic logical representation may define one or morevariables for which text may be generated.

[0039] Based on the semantic frame that the schematic logicalrepresentation uses, the variables for an assessment item may be limitedin their vocabulary (i.e., only a limited number of word choices may beavailable for each variable). For instance, in a Limousine variant of adistance-rate-time semantic frame, vehicle and driver variables may beconstrained to be a limousine and a chauffeur, respectively. Anexemplary schematic logical representation according to an embodiment ofthe present invention may represent the sentence “A car drove 600 miles”logically as “VEHICLE|W TRAVEL(DISTANCE=“600”|P).” In the schematiclogical representation, lexical elements (VEHICLE, TRAVEL, and DISTANCE)may be represented as abstract labels, which are filled in by choosing asemantic frame and substituting appropriate lexical and grammaticalmaterials. In addition, variables (|W and |P) and numeric values, suchas the specification of distance, may permit the schematic logicalrepresentation to abstract itself from language-specific details ofverbal expression while still providing a detailed specification of thecontent and the order in which it is to be presented. Each lexicalelement may represent a variable that ranges over a class of elements.The potential values for a lexical element may depend upon the selectedsemantic frame.

[0040] The term “mental model,” as used herein, refers to a structurewith a limited number of component entities and relationships, subjectto direct imagination and inspection. For example, assessment itemstypically vary along the following dimensions (exemplary values fordistance-rate-time assessment items are provided):

[0041] The number of events (one or two, typically)

[0042] The semantic frame(s) associated with each event

[0043] The primary participants in each event (variations includeoperator v. passenger and the type of vehicle)

[0044] The identity of the primary participants (participants may be thesame or different for each event and each participant may use the samevehicle or a different vehicle)

[0045] The secondary roles relevant to each event (e.g., distance, rate,time, route followed, start point, end point, start time and end time)

[0046] The identity of the secondary roles across events (i.e.,specifying relationships among events)

[0047]FIG. 1 depicts an exemplary high-level implementation of a systemfor performing natural language generation according to an embodiment ofthe present invention. In the embodiment, three primary components areidentified: the document structure generator 105, the logical schemagenerator 115, and the sentence generator 125. The document structuregenerator 105 receives as input a set of abstract choices from a userinterface and produces, for example, an XML document 110. The XMLdocument 110 may represent the underlying schematic content of anassessment item that is being generated. In an embodiment, the documentstructure generator 105 may perform, for example, the following tasks:building a mental model, outlining a document structure which expressesthe mental model in a format compatible with the task-relevant problemstructure, and passing choices about low-level wording to later modules.Building a mental model may include, for example, identifying whichhigh-level semantic frames are to be used, setting up a list of events,and binding variables across events. Outlining a document structure mayinclude, for example, determining the number of sentences required toexpress different parts of the mental model, determining the function ofeach sentence (e.g., stating a set of facts, querying a particularvariable, and the like), and determining whether information isexpressed in each sentence. Passing on the choices may include, forexample, packaging the information for transmission to the logicalschema generator 115.

[0048] In an embodiment, the first intermediate representation 110 maybe an XML document. The first intermediate representation 110 mayspecify the possible values for variables pertinent to the assessmentitem, indicate high-level items pertaining to the assessment itemstructure, and outline the sequence of sentences that will constitutethe final assessment item. An exemplary first intermediaterepresentation 110 is shown in Table 1. TABLE 1 An XML Word ProblemSpecification <?xml><body> <variables>W,X,Y,Z,VONE</variables> <frameid=“1” frameID=“ONE” type=“DRT” VONE=“VEHICLE”/> <event id=“2”frameID=“ONE” eventID=“A” type=“DRT” subj=“W@” rate=“30” rateVar=“Y” /><event id=“3” frameID=“ONE” eventID=“B” type=“DRT” subj=“X@” rate=“70”rateVar=“Z” /> <bindings id=“4” frameID=“ONE” type=“DRT” W=“VONE”X=“VONE” /> <proposition id=“2” frameID=“ONE” eventID=“A”role=“QCColumnA” type=“DRT” distance=“QUERY” rate=“EXPRESSED”time=“UNKNOWN” /> <proposition id=“F” frameID=“ONE” eventID=“B”role=“QCColumnB” type=“DRT” distance=“QUERY” rate=“EXPRESSED”time=“UNKNOWN” /> </body>

[0049] The information contained in Table 1 specifies that fivevariables are used (i.e., W, X, Y, Z, and VONE) and that the sentencesin the resulting assessment item will use vocabulary from a singlesemantic frame of type DRT (distance-rate-time). The variable VONE isdefined to be of type VEHICLE within the semantic frame. In addition,two events are described in the document, with different rate values anddifferent subject variables. Moreover, the resulting document includestwo sentences. Each sentence queries the DISTANCE variable and providesRATE information. Neither sentence defines any information regardingTIME. Furthermore, the first sentence is placed in column A of a wordproblem having a Quantitative Comparison format, and the second sentenceis placed in column B under the same format.

[0050] Although the content of the first intermediate representation 110is abstract, the representation may indicate, in a concise form, theinformation that is to be presented and the organization of theinformation. The succeeding modules may define further details for theresulting assessment item.

[0051] The logical schema generator 115 may perform, for example, twodiscrete tasks. First, the logical schema generator 115 may structuresentences in the resulting assessment item so that the sentences aredirectly tied to the appropriate semantic frame. For example, for adistance-rate-time problem, the logical schema generator 115 mayrestructure the input from the first intermediate representation 110 sothat a series of schematic statements or questions about VEHICLEs,TRAVELing, RATE, TIME and DISTANCE are produced.

[0052] In addition, the logical schema generator 115 may decide theformat of the information contained in each sentence. Particularly, thismay include the type of verb to use and the ordering of elements. Thelogical schema generator 115 may produce a predicate calculus in whichthe arguments are specified by the underlying mental model and in whichthe predicates are variables to be filled by actual natural languageexpressions. However, the logical schema generator 115 may not engage infine details regarding the phrasing of each sentence. Instead, thetransformation of the first intermediate representation 110 to thesecond intermediate representation 120 may be based on detailedinformation about the semantic frames relevant to each problem type. Inan embodiment, the language for the assessment item is stored in thesecond intermediate representation 120. The second intermediaterepresentation 120 may result in an output divided into two sections.The first section may include a series of variable definitions, whichidentify the classes of vocabulary to be used whenever a particularvariable occurs in the text. The second section may include a logicalformula using these variables, which sets up a schema or template forthe resulting assessment item.

[0053] In an embodiment, the second intermediate representation 120 maybe an XML document. The XML document may specify variable definitionsincluding a series of tags for indicating how particular logicalexpressions map to actual vocabulary. Exemplary mappings are shown inTable 2. TABLE 2 Logical Expression Mappings <frameset id=“DRT”type=“9998” /> <frameset id=“TEMP” type=“10002” /> <vardef name=“DRIVE”id=“DRT” frameRole=“27” /> <vardef name=“VEHICLE” id=“DRT” frameRole=“3”/> <vardef name=“TIME” id=“TEMP” frameRole=“2” unit=“5” />

[0054] The definitions in Table 2 may define two semantic frames byreference to each semantic frame's location in a database containing theequivalent of a dictionary specification of all semantic frame elements.The variable DRIVE may be defined to use vocabulary from role 27 in theDRT semantic frame definition, the variable VEHICLE may be defined touse vocabulary from role 3 in the DRT semantic frame definition, and thevariable TIME may be defined to use vocabulary from role 2 in the TEMPsemantic frame definition. Based on these and other variabledefinitions, the remainder of the output may include a precise schematicspecification of the content to be output. Table 3 shows a specificationinstance. TABLE 3 Logical Schema Representation <given> ((W and Xeach)|SB :: PERSON|SB DRIVE(SEPARATE : VEHICLE|OB, SAME : DISTANCE|DI,SAME : NUMBER : TIME|TI) </given> <given> PERSON|W LEAVEONE(ORIGIN|I,CLOCKTIME|M) and PERSON|W ARRIVEONE(DESTINATION|K, CLOCKTIME|O),DRIVEONE((VONE|Y :: VEHICLE|Y), (METER=“50”|S :: DISTANCE|S), STRAIGHT :ROUTE|G) </given> <columnA> AVERAGE : NUMBER : RATE :: PERSON|XDRIVEONE((VONE|Z :: VEHICLE|Z), (TI|U :: TIME|U), EVENT|F :: PERSONTRAVELONE|EVENT((DI|S :: DISTANCE|S), STRAIGHT : ROUTE|H, ORIGIN|J,DESTINATION|L)) RANGE(START|N, END|P) </columnA> <columnB> RATE=“3”</columnB>

[0055] The representation in Table 3 varies from standard logicalrepresentations in order to make the translation to actual naturallanguage text more straightforward. The translation is made morestraightforward by, for example, separating the subject from thepredicate; treating variables, such as |N, |S and |P, as labels attachedto the elements that are verbally expressed; by treating modifiers asbeing attached to the argument that they modify; and the like.

[0056] The sentence generator 125 may translate the logicalrepresentations in the second intermediate representation 120 intoassessment item text. The sentence generator 125 may parse the logicalrepresentations, annotate the resulting parse tree with grammaticalinformation, find words and word forms in a dictionary based related toa selected language to fit the grammatical information and output theresulting text in order to complete the translation.

[0057] The sentence generator 125 may cover a wide range of linguisticdetails that are generally unrelated to the substance of the assessmentitem. For example, the sentence generator 125 may record the number oftimes an entity has been mentioned so that the choice of determinersbetween “a”/“an” and “the” is appropriate. Other grammatical phenomenamay also be tracked, such as subject/verb agreement, preposition andcase marking, and the like.

[0058] The output representation 130 may be an XML document having alltext converted to the specified language. The output representation 130may retain XML tags identifying the role that each text chunk performsin the structure of the assessment item. The output representation 130may then be formatted for display and presented as an assessment item.The resulting assessment item may be used in connection with a testcreation system.

[0059]FIGS. 2-6 depict exemplary screen shots of a graphical userinterface used to generate assessment items according to an embodimentof the present invention. In the embodiment disclosed in FIGS. 2-6, adistance-rate-time assessment item is generated. Other embodimentsinclude interest accumulation problems, tax problems, problemsassociated with producing items and the like. The variables used in eachembodiment may be selected based on the problem type to be solved.Accordingly, the embodiment described below and in reference to FIGS.2-6 is not meant to be limiting, but merely exemplary of the generationof one type of an assessment item.

[0060] Generation of the distance-rate-time assessment item may besub-divided into five tasks: assigning mental model structure variables,defining identity variables in the mental model structure, determining atask-relevant problem structure, describing a document format andstructure, and determining language variations.

[0061] The assignment of mental model structure variables may includedefining the number of events; the number of distinct frames; the typesof participants; the type of trip; the default unit of measure; and theactual distance, rate and time units used in each event. In anembodiment, a distance-rate-time problem has one or two events.Alternately, more events may be used for a distance-rate-time problem.If two or more events are used, the semantic frame for each event maydiffer. For example, a speed of a truck may be at issue in a firstevent, and a speed of a train may be at issue in a second event.Alternately, the semantic frames for each event may be the same. Thetype of participant may be used to specify whether people and/orvehicles are mentioned in the assessment item. If more than one event isincluded in the assessment item, the primary participants may includedifferent people using the same vehicle or different vehicles, or thesame person using the same vehicle or different vehicles. The trip typemay be used to determine the values of one or more secondary variables.The trip type may include a round trip, a one way trip, two people orvehicles heading towards each other, two people or vehicles heading awayfrom each other, and numerous other types of trips.

[0062] Defining the identity variables in the mental model structure mayinclude choosing variables to express one or more of the following ineach event: people, vehicles, time, distance, rate, starting locations,ending locations, starting times, ending times, the route and the eventas a whole. While the distance, rate and time values are central todetermination of the problem, other variables may be purely incidentalto the determination of the answer to the problem.

[0063] Determination of a task-relevant problem structure may includedetermining which variable is being determined for each event, theanswer for each event, numeric values for the mathematical variables(i.e., numeric values for distance, rate and time in each event),whether some information is provided in summary form, and whether timeis expressed as a sequence of clock times. Most problems within aTransportation semantic frame may solve the equation d=rt. In such acase, the equation may map to the semantic roles Distance, Rate andTime. Another problem type that may be within a Transportation semanticframe may include a fuel efficiency problem. In such a problem, theequation to solve may be f=ed, where f is fuel consumption, e is fuelefficiency, and d is distance. If a unique answer is required forsolution to an assessment item, only one variable may be unspecified.Accordingly, values may be assigned to all but one variable to providefor a unique solution. Implicit values may be assigned to complicate anassessment item. For example, the amount of time may not be explicitlyspecified. Rather, the assessment item may state that the person/vehicletraveled from 3:00 to 6:00.

[0064] Describing a document format and structure may includedetermining the format of the problem (i.e., quantitative comparison vs.problem solving) and the arrangement of content between the options andthe stem. The problem format may determine the type of language used forthe query of the assessment item. For example, a problem-solving(multiple choice) assessment item may query information in the form of aquestion: “Which of the following?” “How much?” or “How many?” Incontrast, a quantitative comparison question may query information in aphrasal form: “The number of hours required . . . ,” “The time taken to. . . ,” “The speed at which . . . ” Moreover, content may be arrangedin different sections of an assessment item. For instance, in aquantitative comparison assessment item, both background information andthe questions may be posed in the columns. Alternatively, backgroundinformation may be posed in a question stem and the questions may beposed in the columns. Such arrangements are merely exemplary. Otherembodiments may arrange information in alternate arrangements withoutlimitation.

[0065] Determining language variations may include, for example,selecting a language, selecting a sentence structure, selecting referentidentification types (identifying people by name or generically,identifying object with labels or by description), selecting detailedphrasing, determining whether quantities are in integer format,determining whether rates are identified as constant or average, andwhether events are described in the present or past tense. The languagemay be selected based on a variable assigned by a user. In anembodiment, different data structures are stored for each language. Thedata structures may include, without limitation, words (in the selectedlanguage) available for use in assessment items and rules pertaining tosentence structure and grammar. The sentence structure may include anactive or passive verb form. Referent identification may include callinga person in an event, for example, “Fred,” “a driver,” “the driver,” or“driver A.”

[0066]FIG. 2 depicts an exemplary screen shot of an existing wordproblem assessment item according to an embodiment of the presentinvention. The menu bar may be used to modify one or more variables forthe assessment item.

[0067]FIG. 3 depicts an exemplary variable modification screen for adistance-rate-time assessment item according to an embodiment of thepresent invention. In an embodiment, the variable modification screenpermits a user to alter the variable for which the assessment itemsolves 305, the basic document format 310, the core participant type315, and the number of events 320.

[0068]FIG. 4 depicts an exemplary detailed control screen for adistance-rate-time assessment item according to an embodiment of thepresent invention. In an embodiment, the detailed control screen permitsa user to alter the number of distinct vehicle types 405, the name usedfor each person 410, the type of trip 415, and numerous other parameterspertaining to the syntactic structure and the secondary grammaticalcontent.

[0069]FIG. 5 depicts an exemplary task-relevant problem structure for anassessment item according to an embodiment of the present invention. Inan embodiment, at least one of the distance, rate and time variables 505and 510 may be set for each event in the assessment item. If desired,the user may assign constant values 515 to the assessment item. Inaddition, an answer 520 to the assessment item may be supplied ifdesired. If an answer is supplied, values may be selected from theranges defined for each event to ensure that the specified answer iscorrect for the resulting assessment item. If any value is unspecified,a random value may be selected for the variable.

[0070]FIG. 6 depicts an exemplary measurement unit definition screenaccording to an embodiment of the present invention. Default units maybe selected for one or more of a rate, a time and a distance eitherglobally 605 or on a per event basis 610 and 615.

[0071] Using the graphical user interface described in FIGS. 2-6, newassessment item instances may be generated from an underlying model.Alternatively, parameters may be altered in an existing assessment itemto generate a new assessment item.

[0072] Although this invention has been illustrated by reference tospecific embodiments, it will be apparent to those skilled in the artthat various changes and modifications may be made which clearly fallwithin the scope of the invention. The invention is intended to beprotected broadly within the spirit and scope of the appended claims.

What is claimed is:
 1. A method for automatically generating a wordproblem, the method comprising: receiving a designation of a problemtype from a user; identifying one or more variables based on the problemtype; assigning at least one value to at least one variable; receiving,from a user, a designation of a variable for which to solve; andgenerating an assessment item of the problem type based on thevariables.
 2. The method of claim 1 wherein identifying one or morevariables comprises identifying a language used to generate theassessment item.
 3. The method of claim 1 wherein the problem typecomprises a mathematical word problem.
 4. The method of claim 3 whereinassigning at least one value to at least one variable comprisesdetermining a unit of measurement for a variable.
 5. The method of claim1 wherein assigning at least one value to at least one variablecomprises assigning a range of values to a variable from which a valuefor the variable is selected when the assessment item is generated. 6.The method of claim 1 wherein assigning at least one value to at leastone variable comprises assigning a variable used to determine a formatfor the assessment item.
 7. A method of automatically generating anassessment item, the method comprising: receiving one or more inputsfrom a user; generating one or more variables based on the one or moreinputs; determining one or more relationships between at least two ofthe variables; and generating an assessment item based on the one ormore variables and the one or more relationships.
 8. The method of claim7 wherein the one or more relationships comprise relationships based onone or more of word order; word choice; word format; sentence structure;grammar; and language.
 9. The method of claim 7 wherein the assessmentitem is a mathematical word problem.
 10. A method of automaticallygenerating an assessment item, the method comprising: receiving one ormore inputs pertaining to the format of an assessment item, wherein theformat of the assessment item comprises at least one event; selectingone or more variables for use in the assessment item; determining arelationship between variables assigned to each event; determining aformat for the assessment item; and generating an assessment item basedon at least the format for the assessment item and the relationshipbetween variables assigned to the at least one event.
 11. The method ofclaim 10 wherein the one or more inputs comprise one or more of thefollowing: a number of events; a number of distinct frames; a type ofparticipant in each event; a type of assessment item to generate; and aunit of measure for each variable.
 12. The method of claim 10 whereindetermining a relationship for the variables assigned to each eventcomprises one or more of the following: determining a variable for whichto solve for each event; determining an answer for each event;determining a value for one or more variables; and determining avariable format.
 13. The method of claim 10 wherein determining a formatfor the assessment item comprises: determining a problem format havingone or more sections; and determining content to place within eachsection.
 14. The method of claim 10 wherein generating an assessmentitem comprises one or more of the following: selecting a language forthe assessment item; selecting a sentence structure for each sentence inthe assessment item; selecting identification types for one or more ofthe variables; determining a numerical format for each of the one ormore variables; and determining a verb tense to use for each event. 15.The method of claim 14 wherein selecting identification types comprisesdetermining to identify a variable denoting a person by using a propername.
 16. The method of claim 14 wherein selecting identification typescomprises determining to identify a variable denoting a persongenerically.
 17. The method of claim 14 wherein selecting identificationtypes comprises determining to identify a variable denoting an object byusing a label.
 18. The method of claim 14 wherein selectingidentification types comprises determining to identify a variabledenoting an object by using a description of the object.
 19. A systemfor automatically generating an assessment item, the system comprising:a processor; and a computer-readable storage medium operably connectedto the processor, wherein the computer-readable storage medium containsone or more programming instructions for performing a method ofautomatically generating an assessment item, the method comprising:receiving a designation of a problem type from a user, determining oneor more variables based on the problem type, assigning at least onevalue to at least one variable, receiving, from a user, a designation ofa variable for which to solve, and generating an assessment item of theproblem type based on the variables.
 20. A system for automaticallygenerating an assessment item, the system comprising: a processor; and acomputer-readable storage medium operably connected to the processor,wherein the computer-readable storage medium contains one or moreprogramming instructions for performing a method of automaticallygenerating an assessment item, the method comprising: receiving one ormore inputs from a user, generating one or more variables based on theone or more inputs, determining one or more relationships between atleast two of the variables, and generating an assessment item based onthe one or more variables and the one or more relationships.
 21. Asystem for automatically generating an assessment item, the systemcomprising: a processor; and a computer-readable storage medium operablyconnected to the processor, wherein the computer-readable storage mediumcontains one or more programming instructions for performing a method ofautomatically generating an assessment item, the method comprising:receiving one or more inputs pertaining to the format of an assessmentitem, wherein the format of the assessment item comprises at least oneevent, selecting one or more variables for use in the assessment item,determining a relationship between variables assigned to each event,determining a format for the assessment item, and generating anassessment item based on at least the format for the assessment item andthe relationship between variables assigned to the at least one event.22. A method of automatically generating an assessment item, the methodcomprising: receiving one or more input parameters; generating adocument structure based on the one or more input parameters; producinga logical schema using the document structure; and generating anassessment item based on the logical schema.
 23. The method of claim 22wherein generating a document structure comprises: building a mentalmodel; and outlining the document structure based on the mental model.24. The method of claim 23 wherein building a mental model comprises:selecting one or more semantic frames; generating a list of one or moreevents; and binding one or more variables across the one or more events.25. The method of claim 23 wherein outlining the document structurecomprises: generating one or more sentences for the mental model;determining a function for each sentence; and determining information toexpress in each sentence.
 26. The method of claim 22, furthercomprising: storing the document structure in a file.
 27. The method ofclaim 26 wherein the file is an Extendable Markup Language (XML) file.28. The method of claim 22 wherein the document structure includes oneor more of the following: one or more variables; one or more values forat least one of the one or more variables; a mental model structure; andan outline of a sequence of one or more sentences for the assessmentitem.
 29. The method of claim 22 wherein producing a logical schemacomprises: outlining a sentence structure for one or more sentences; anddetermining an information format for each sentence.
 30. The method ofclaim 29 wherein determining an information format comprises one or moreof the following: determining a verb type for each sentence; determiningan ordering of one or more elements for each sentence; and determiningone or more vocabulary sets to use for each element.
 31. The method ofclaim 22, further comprising: storing the logical schema in a file. 32.The method of claim 31 wherein the file is an XML file.
 33. The methodof claim 22 wherein generating an assessment item comprises: parsing thelogical schema; annotating the parsed logical schema with grammaticalinformation; determining words and word forms based on the grammaticalinformation; and outputting text representing the assessment item. 34.The method of claim 22 wherein an input parameter determines a languageused for the assessment item.
 35. A method of automatically generatingan assessment item, the method comprising: assigning one or more mentalmodel structure variables; defining one or more identity variables for amental model structure; determining a task-relevant problem structure;defining a document format; and determining language variations.
 36. Themethod of claim 35 wherein assigning one or more mental model structurevariables comprises defining one or more of the following: one or moreevents; one or more distinct semantic frames; one or more participanttypes; and an event type for each event.
 37. The method of claim 36wherein determining a task-relevant problem structure comprises:determining a variable for which to solve for each event; determining ananswer for each event; and determining one or more values for eachvariable.
 38. The method of claim 35 wherein determining languagevariations comprises: selecting a sentence structure for each of one ormore sentences; selecting a referent identification type for each of oneor more participants; and determining a tense for each of one or moreevents.
 39. The method of claim 35, further comprising: assigning avariable determining a language in which to generate the assessmentitem.
 40. A system for automatically generating an assessment item, thesystem comprising: a processor; and a computer-readable storage mediumoperably connected to the processor, wherein the computer-readablestorage medium contains one or more programming instructions forperforming a method of automatically generating an assessment item, themethod comprising: receiving one or more input parameters, generating adocument structure based on the one or more input parameters, producinga logical schema from the document structure, and generating anassessment item from the logical schema.
 41. A system for automaticallygenerating an assessment item, the system comprising: a processor; and acomputer-readable storage medium operably connected to the processor,wherein the computer-readable storage medium contains one or moreprogramming instructions for performing a method of automaticallygenerating an assessment item, the method comprising: assigning one ormore mental model structure variables, defining one or more identityvariables for a mental model structure, determining a task-relevantproblem structure, defining a document format, and determining languagevariations.